Skip to content

Feature: Notion Data Source #100

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 9 commits into
base: main
Choose a base branch
from
Open

Feature: Notion Data Source #100

wants to merge 9 commits into from

Conversation

@anaselmhamdi
Copy link

@anaselmhamdi anaselmhamdi commented Jun 6, 2023 •
edited
Loading

Notion data source

  • The Data Source syncs all pages and children blocks that the integration has been allowed access to
  • I do a very simple parsing where I parse all the rich_text blocks' plain_text to create paragraph to index
  • I had to remove the line from parsers.pdf import split_PDF_into_paragraphs as it wouldn't let me run the app otherwise. It looked like the line wasn't used anywhere though
  • I have no issues running the app and indexing documents

image

image

Let me know if you want me to add more details for the Notion integration setup, I can also embed a Loom video if you guys wish.

Changes

  • Added notion.py data source
  • Added Notion config in UI
  • Added Notion logo
  • Removed (orphan ?) from parsers.pdf import split_PDF_into_paragraphs in app/indexing/index_documents.py

UI Screenshots

Data panel:
image
Data source setup:
image
Actual search result:
image

@anaselmhamdi anaselmhamdi marked this pull request as draft June 6, 2023 13:24
@anaselmhamdi anaselmhamdi changed the title Feature/notion Feature: Notion integration Jun 6, 2023
@anaselmhamdi anaselmhamdi changed the title Feature: Notion integration Feature: Notion Data Source Jun 6, 2023
@anaselmhamdi anaselmhamdi marked this pull request as ready for review June 6, 2023 13:37
@yuvalsteuer
Copy link
Contributor

Hi this is great! have you thought or adding a notion element for ticket/issues. we have a prebuilt UI element that might go well.

@yuvalsteuer yuvalsteuer requested a review from Roey7 June 6, 2023 14:59
@yuvalsteuer
Copy link
Contributor

yuvalsteuer commented Jun 6, 2023 •
edited
Loading

can you take pictures of the different UI elements?

  • During the setup stage.
  • Datasource setup.
  • Actual search result.
    etc...

Thanks for taking the time and contributing to this project. you're awesome.

@anaselmhamdi
Copy link
Author

Sure, I'll do it this week !

@anaselmhamdi
- added all database pages
3c371c8 
- added more retries
- handling of potential errors stopping the indexing process
@anaselmhamdi
Copy link
Author

anaselmhamdi commented Jun 13, 2023 •
edited
Loading

@yuvalsteuer I'm back and:

  • I added screenshots of the UI with the elements you talked about => I edited the intial PR comment
  • I also added logic to index all the database related pages
  • I've seen the UI element but i was thinking of maybe doing it in another PR

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Roey7 Roey7 Awaiting requested review from Roey7

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@anaselmhamdi @yuvalsteuer